Mixed reality image processing apparatus and mixed reality image processing method

ABSTRACT

There is provided a mixed reality image processing apparatus capable of forming mixed reality image data that matches an illumination environment of an external world. The mixed reality image processing apparatus includes a standard illumination environment processing unit configured to extract illumination environment information, which indicates the illumination information of the external world, from image data imaged by an imaging unit, and a local illumination environment processing unit configured to convert mixed reality image data, which is formed by combining virtual image data to the image data, into image data corresponding to the illumination environment of the external world based on the illumination environment information.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to a technique for generating mixedreality image data by combining virtual image data to captured imagedata.

Description of the Related Art

In recent years, a mixed reality technique, i.e., an MR technique hasbeen known as a technique for mixing a real world and a virtual worldseamlessly in real time. As one type of the MR technique, there is knowna technique that uses a video see-through head mounted display (HMD) toallow an HMD user to observe a mixed reality image. The mixed realityimage is created by imaging an object that substantially coincides withan object observed from a line-of-sight position of the HMD user withuse of a video camera and the like, and displaying this captured imagedata with computer graphics (CG) superimposed thereon.

An imaging unit mounted on the video see-through HMD capturesobservation image data in an external world that substantially coincideswith the line-of-sight position of the HMD user. The imaging unitincludes two pairs of an image sensor and an optical system for a righteye and a left eye for generating stereo image data, and a digitalsignal processor (DSP) for performing image processing. A display unitdisplays the mixed reality image data created by combining CG to theobject that substantially coincides with the object observed from theline-of-sight position of the HMD user. Further, the display unit isconfigured to deal with a pair of images of the right side and the leftside like the imaging unit, and includes two pairs of a display deviceand an optical system for the right eye and the left eye.

A relationship between the external world and the mixed reality imagedata will be described now. The above-described display unit isconfigured to display image data input from an external apparatuswithout making any adjustment thereto. Therefore, even when the externalworld is at dust, the external world is at noon so that it is extremelybright, or the external world is under a slightly dark environment inthe shade of a tree, generally, the mixed reality image data displayedon the display unit is image data unaffected by the environment of theexternal world around the HMD user, in which a brightness and a colorare uniformly adjusted. Therefore, a gap may be generated betweenbrightness and color sensations felt by the HMD user to the ambientenvironment, and the actual environment of the external world.

This influence cannot be also ignored for an optical see-through HMD.When CG data is superimposed on a see-through image of the externalworld in the optical see-through HMD, the brightness and the color areuniformly adjusted only in the CG data, whereby the balance and thecolor are unbalanced between the CG data and the see-through image ofthe external world. This leads to such a problem that the CG data mayhave an unnatural brightness and color to the see-through image of theexternal world depending on the environment of the external world,thereby impairing a realistic sensation.

Under this circumstance, Japanese Patent Application Laid-Open No.2002-244077 discusses a technique that reduces a shutter speed of animaging unit and increases a lighting time of an illumination lightsource of a display unit under a bright environment, while increasingthe shutter speed and reducing the lighting time of the illuminationlight source under a dark environment. With this adjustment, thetechnique discussed in Japanese Patent Application Laid-Open No.2002-244077 aims at maintaining the display brightness of the displayunit at a brightness that matches see-through light even under variouskinds of environments of an external world. Further, the techniquediscussed in Japanese Patent Application Laid-Open No. 2002-244077 aimsat imaging an object while maintaining the brightness within a certainrange by changing the shutter speed of the imaging unit according to thebrightness of the external world.

Japanese Patent No. 03423402 discusses a technique that detects a colortemperature of an external world by a sensor, and adjusts a colorbalance of display image data to be displayed on a display unitaccording to the detected color temperature. With this adjustment, thetechnique discussed in Japanese Patent No. 03423402 aims at generatingdisplay image data less unbalanced with see-through light.

However, the techniques discussed in Japanese Patent ApplicationLaid-Open No. 2002-244077 and Japanese Patent No. 03423402 aretechniques for adjusting the brightness and color of the display imagedata according to an illumination environment of the external world, andcannot display mixed reality image data that matches the illuminationenvironment of the external world on the video see-through HMD.

SUMMARY OF THE INVENTION

According to an aspect of the present invention, a mixed reality imageprocessing apparatus includes an extraction unit configured to extractillumination environment information, which indicates an illuminationenvironment of an external world, from image data imaged by an imagingunit, and a conversion unit configured to convert mixed reality imagedata, which is formed by combining virtual image data to the image data,into image data corresponding to the illumination environment of theexternal world based on the illumination environment information.

According to the present disclosure, it is possible to generate themixed reality image data that matches the illumination environment ofthe external world.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a functional configuration of amixed reality system using a video see-through HMD according to a firstexemplary embodiment.

FIG. 2 is a block diagram illustrating a detailed configuration of andaround a standard illumination environment processing unit and a localillumination environment processing unit in the video see-through HMDaccording to the first exemplary embodiment.

FIG. 3 illustrates a graph indicating a corresponding relationshipbetween Red-Green-Blue (RGB) signal intensities in an RGB image formatand a color temperature.

FIG. 4 is a flowchart illustrating processing performed by the standardillumination environment processing unit in the video see-through HMDaccording to the first exemplary embodiment.

FIG. 5 is a flowchart illustrating a flow of image data in the localillumination environment processing unit 107 according to the firstexemplary embodiment.

FIG. 6 is a flowchart illustrating timing adjustment processingperformed by the local illumination environment processing unit 107 inthe video see-through HMD according to the first exemplary embodiment.

FIG. 7 is a time chart indicating a temporal relationship between a flowof image data and a flow of information in local illuminationenvironment processing according to the first exemplary embodiment.

FIGS. 8A, 8B, 8C, and 8D illustrate tables indicating data structures ofillumination environment information.

FIGS. 9A, 9B, 9C, and 9D illustrate what kind of characteristics animaging device and a display device have.

FIG. 10 is a block diagram illustrating a functional configuration of amixed reality system using a video see-through HMD according to a secondexemplary embodiment.

FIG. 11 is a block diagram illustrating a further detailed configurationof the video see-through HMD according to the second exemplaryembodiment.

FIG. 12 is a time chart indicating a temporal relationship between aflow of image data and a flow of information in image combiningprocessing and local illumination environment processing according tothe second exemplary embodiment.

FIGS. 13A and 13B are block diagrams illustrating a modification of themixed reality system according to the second exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the inventionwill be described in detail below with reference to the drawings.

First, a first exemplary embodiment of the present invention will bedescribed. FIG. 1 illustrates a functional configuration of a mixedreality system using a video see-through head mounted display (HMD) 101according to the first exemplary embodiment. The mixed reality systemillustrated in FIG. 1 includes the video see-through HMD 101 accordingto the first exemplary embodiment. The video see-through HMD 101includes an imaging unit 103, a standard illumination environmentprocessing unit 104, an image output unit 105, an image input unit 106,a local illumination environment processing unit 107, and a display unit108. The mixed reality system is configured to use the video see-throughHMD 101 as an example of a mixed reality image processing apparatus.

The imaging unit 103 captures image data (hereinafter referred to ascaptured image data) of an external world that substantially coincideswith a line-of-sight position of an HMD user. The imaging unit 103includes two pairs of image sensor and optical system for a right eyeand a left eye for generating stereo image data, and a DSP forperforming image processing. A solid-state image sensor, which isrepresented by a charge coupled device (CCD) image sensor and acomplementary metal-oxide semiconductor (CMOS) image sensor, is used foreach of the image sensors.

The display unit 108 displays mixed reality image data (MR image data)formed by combining the captured image data and CG. The display unit 108is configured to deal with images of the right side and the left sidesimilar to the imaging unit 103, and includes two pairs of displaydevice and optical system for the right eye and the left eye. A smallliquid-crystal display or a retina scan type device using Micro ElectroMechanical Systems (MEMS) is used for each of the display devices.

The image output unit 105 converts a format of output image dataaccording to an interface between the apparatuses. The image input unit106 converts a format of input image data according to an interfacebetween the apparatuses. A method that can meet a requirement of areal-time capability and can transmit a large amount of data is used foreach of interfaces of the image output unit 105 and the image input unit106. Examples of such a method include a metallic cable such as auniversal serial bus (USB) or Institute of Electrical and ElectronicsEngineers (IEEE) 1394, and an optical fiber such as Gigabit Ethernet(registered trademark).

Referring to FIG. 1, the mixed reality system includes an imageprocessing apparatus 102. The image processing apparatus 102 includesthe image output unit 105, the image input unit 106, aposition/orientation measurement unit 109, a captured image storage unit110, a content database (DB) 111, a CG drawing unit 112, and an imagecombining unit 113. The image processing apparatus 102 can be embodiedby an information processing apparatus that has a high-performancecalculation processing function and graphics generation/displayfunction, such as a personal computer (PC) and a workstation.

The position/orientation measurement unit 109 measuresposition/orientation information that indicates at least any one of aposition and an orientation of the HMD user. More specifically, theposition/orientation measurement unit 109 extracts a marker and/or anatural feature from the captured image data input from the image inputunit 106, and measures the position/orientation information using them.The measured position/orientation information is output as necessary tothe CG drawing unit 112 to be used for a calculation of a shape ofvirtual image data (CG) to be drawn.

The content DB 111 is a database that stores contents of virtual imagedata. The CG drawing unit 112 draws virtual image data based on theposition/orientation information measured by the position/orientationmeasurement unit 109 and the contents stored in the content DB 111. Thecaptured image storage unit 110 stores the captured image data inputfrom the image input unit 106. The image combining unit 113 combines thecaptured image data and the virtual image data. It is desirable that thecaptured image data on which the virtual image data is to besuperimposed is captured image data from which the position/orientationinformation for drawing the virtual image data is detected. However, ifthe real-time capability is impaired due to a transmission delay betweenthe systems, a processing time required to draw CG, and the like, themixed reality system may be configured to superimpose the virtual imagedata onto latest updated captured image data at a timing when thevirtual image data is generated. Using a predicted value as theposition/orientation information for CG drawing at this time can reducea time lag between the captured image data and the virtual image data.

Based on this configuration, a flow of image data will be brieflydescribed. The captured image data output from the imaging unit 103 isoutput to the image processing apparatus 102 and is stored into thecaptured image storage unit 110, after being processed by the standardillumination environment processing unit 104. Then, virtual image datais superimposed on the captured image data stored in the captured imagestorage unit 110 by the image combining unit 113. As a result, mixedreality image data is generated. The generated mixed reality image datais output to the video see-through HMD 101 and is processed by the localillumination environment processing unit 107. Then, the mixed realityimage data is displayed on the display unit 108.

FIG. 2 illustrates a detailed configuration of and around the standardillumination environment processing unit 104 and the local illuminationenvironment processing 107 in the video see-through HMD 101.

The imaging unit 103 captures captured image data so as to maintain anactual appearance as much as possible. If the external world is dark, adark image is captured as the captured image data. If the external worldis bright, a bright image is captured as the captured image data.Further, if a color temperature is high in the external world, thecaptured image data also has a high color temperature. If the colortemperature is low in the external world, the captured image data alsohas a low color temperature. In this manner, the imaging unit 103performs imaging processing capable of outputting captured image datathat matches the external world as much as possible. This does not meanthat image data formed by performing an analog-digital (AD) conversionon an analog signal photoelectrically converted by the imaging devicewithout any arrangement made thereto is output as the captured imagedata that matches the external world. It is desirable to reduce ananalog noise by guiding a large amount of light to the imaging device,and correct a luminance by an imaging device characteristic correctionunit 201, which will be described below. Further, it is desirable toalso correct a color due to a color filter and the like. The imagingunit 103 performs processing that enables the environment of thecaptured image data to match the external world as the whole unit. Theimaging unit 103 includes the imaging device characteristic correctionunit 201. The imaging device characteristic correction unit 201 correctsa characteristic depending on an individual imaging device.

As illustrated in FIG. 2, the standard illumination environmentprocessing unit 104 includes an illumination environment informationextraction unit 202, an image identification information source 203, aluminance conversion unit 204, and a color conversion unit 205. Theillumination environment information extraction unit 202 extractsinformation that indicates an illumination environment of the externalworld (hereinafter referred to as illumination environment information),such as color temperature information and luminance information, fromthe captured image data by performing image processing. Then, theillumination environment information extraction unit 202 outputs theextracted illumination environment information together with imageidentification information contained in the image identificationinformation source 203 to the local illumination environment processingunit 107. The luminance information here means, for example, informationthat indicates a Y signal of the captured image data in a YUV format.The color temperature information means, for example, information thatindicates a color temperature calculated from a ratio among RGB signalintensities of the captured image data in an RGB format. In addition tothe above-described processing, the illumination environment informationextraction unit 202 embeds the image identification information into thecaptured image data by an electronic watermark technique, and outputs itto the luminance conversion unit 204. The image identificationinformation only has to be added as an attribute of the captured imagedata. Therefore, the image identification information may be added to aheader of image data, and the image identification information may becombined or added by a different method depending on a systemconfiguration.

The image identification information source 203 is a collection of imageidentification information. The image identification information onlyhas to be at least information that allows identification of image datafrom the time when captured image data is captured to the time whenvirtual image data is superimposed onto this captured image data.Therefore, if a time period corresponding to 10 frames is required froma capture of captured image data to superimposition of virtual imagedata onto this captured image data, only image identificationinformation corresponding to the 10 frames should be prepared, so thatthe image identification information can be expressed by 4 bits. In thismanner, the image identification information only has to be informationthat allows image data to be uniquely identified. For example, as simplemethods, the image identification information can be realized by merelynumbering captured image data, or can be also realized as a time stampof a time when captured image data is captured. Actually, how to realizethe image identification information and an amount of the informationare determined in consideration of factors such as a variation in thetime period from a capture of captured image data to superimposition ofvirtual image data onto this captured image data, and a possibility thatthe image identification information is also used as information thatindicates a date/time when the captured image data is captured.

The luminance conversion unit 204 converts a luminance of the capturedimage data. The luminance conversion unit 204 adjusts, for example, theY signal of the captured image data in the YUV format. If the Y signalis expressed by 8 bits, i.e., within a decimal range of 0 to 255, forexample, the Y signal is adjusted to around 127, which is a middlevalue. At that time, the Y signal is adjusted so that each of RGB valuesis contained in a domain after conversion into the RGB format. If eachof RGB signals is expressed by 8 bits, and the following RGB-YUVconversion equation is used, the luminance conversion unit 204 adjuststhe Y signal so that each of the RGB signals is contained within thedecimal range of 0 to 255.R=Y+1.40200VG=Y−0.344114U−0.71414VB=Y+1.77200U

Desirably, a value that makes the captured image data not too bright andnot too dark, and visually comfortable is selected as an adjustmentvalue of the Y signal, and this value is determined in consideration ofthe captured target, the environment under which the captured image datais observed, the application, and the like. The format of the image datais converted if necessary, before and after the luminance conversion.The luminance conversion processing here means a brightness adjustmentperformed on a digital signal level.

The color conversion unit 205 converts a color temperature of thecaptured image data. The color conversion unit 205 makes an adjustmentso that, for example, a ratio among RGB signal intensities becomes 1:1:1in the captured image data in the RGB format. The adjusted ratio amongRGB signals may be a ratio corresponding to a color temperature definedby a standard such as standard RGB (sRGB) selected in consideration ofthe environment under which the captured image data is observed, theapplication, and the like. A method for calculating the colortemperature from the ratio among RGB signal intensities will bedescribed below with reference to FIG. 3. The format of the capturedimage data is converted if necessary, before and after the colorconversion. The color conversion by the color conversion unit 205 meansa white balance adjustment performed on the digital signal level.

As illustrated in FIG. 2, the local illumination environment processingunit 107 includes an illumination environment information processingunit 206, an image identification information extraction unit 207, atiming adjustment unit 208, an image storage unit 209, a luminanceconversion unit 210, and a color conversion unit 211.

The illumination environment information processing unit 206 includes astorage unit (not illustrated) for storing illumination environmentinformation, and sequentially stores therein the illuminationenvironment information input from the illumination environmentinformation extraction unit 202. Further, image identificationinformation is input from the timing adjustment unit 208 into theillumination environment information processing unit 206, by which theillumination environment information processing unit 206 reads outillumination environment information corresponding to this imageidentification information, and outputs a readout success signal(Acknowledgement (ACK)) to the timing adjustment unit 208. In additionthereto, the illumination environment information processing unit 206outputs the image identification information and the luminanceinformation contained in the illumination environment information to theluminance conversion unit 210, and outputs the image identificationinformation and the color temperature information contained in theillumination environment information to the color conversion unit 211.

The image identification information extraction unit 207 extracts theimage identification information from the mixed reality image data. Thisimage identification information is the image identification informationembedded in the captured image data by the illumination environmentinformation extraction unit 202. The image identification informationextracted from the mixed reality image data is output to the timingadjustment unit 208. Further, the image identification informationextraction unit 207 bundles the mixed reality image data and theextracted image identification information together as one set, andstores them into the image storage unit 209.

The timing adjustment unit 208 outputs the image identificationinformation input from the image identification information extractionunit 207 to the illumination environment information processing unit206. The illumination environment information processing unit 206outputs a readout success signal (ACK) to the timing adjustment unit208, if succeeded in reading out illumination environment informationcorresponding to this image identification information. The timingadjustment unit 208 adjusts a timing of reading out the mixed realityimage data from the image storage unit 209, after confirming the ACK.More specifically, the timing adjustment unit 208 adjusts the timing sothat the mixed reality image data is read out from the image storageunit 209 at a timing at which it becomes possible to perform processingusing the read illumination environment information by the luminanceconversion unit 210. This adjustment allows the luminance conversionunit 210 and the color conversion unit 211 to sequentially perform theluminance conversion and the color conversion using the illuminationenvironment information with a delay as short as possible.

The image storage unit 209 includes a line buffer or a frame buffer thatstores the mixed reality image data. The image storage unit 209 storesthe mixed reality image data together with the image identificationinformation extracted by the image identification information extractionunit 207. The luminance conversion unit 210 converts a luminance of themixed reality image data. The content of the processing is similar tothe luminance conversion of captured image data by the luminanceconversion unit 204, but the luminance information in the illuminationenvironment information is used as an adjustment value therefor. Theluminance conversion unit 210 can also check an error by reading out theimage identification information together with the mixed reality imagedata from the image storage unit 209, and comparing it with the imageidentification information of the illumination environment information.As a result of this luminance conversion, the luminance of the mixedreality image data becomes substantially equivalent to the luminance ofthe captured image data before the conversion processing performed bythe luminance conversion unit 204.

The color conversion unit 211 converts a color temperature of the mixedreality image data. The content of the processing is similar to thecolor conversion of captured image data by the color conversion unit205, but the color conversion unit 211 adjusts RGB signals based on thecolor temperature information in the illumination environmentinformation. The color conversion unit 211 can also check an error byinputting the image identification information together with the mixedreality image data from the luminance conversion unit 210, and comparingit with the image identification information of the illuminationenvironment information. As a result of this color conversionprocessing, the color temperature of the mixed reality image databecomes substantially equivalent to the color temperature of thecaptured image data before the conversion processing performed by thecolor conversion unit 205.

The display unit 108 displays the input mixed reality image data whilemaintaining its brightness and tint. If the input mixed reality imagedata is dark, dark HMD observation image data is displayed. If the inputmixed reality image data is bright, bright HMD observation image data isdisplayed. Further, if the input mixed reality image data has a highcolor temperature, displayed HMD observation image data also has a highcolor temperature. If the input mixed reality image data has a low colortemperature, displayed HMD observation image data also has a low colortemperature. The display unit 108 performs processing so as to restraina characteristic specific to the display device by a display devicecharacteristic correction unit 212, which will be described below, andconverts the mixed reality image data input into the display unit 108into HMD observation image data. The display device 108 includes thedisplay device characteristic correction unit 212. The display devicecharacteristic correction unit 212 corrects a characteristic dependingon an individual display device.

FIG. 3 illustrates a graph indicating a corresponding relationshipbetween RGB signal intensities in the RGB image format and a colortemperature. A color temperature 6500K is set as a reference, and an RGBsignal gain is adjust so that an RGB signal ratio becomes 1:1:1 when awhite plate is imaged under the color temperature 6500K. This isreferred to as a white RGB signal gain under the reference colortemperature 6500K. FIG. 3 illustrates the relationship of RGB signalintensities to each color temperature when the same white plate isimaged under various color temperatures, while maintaining this RGBsignal gain. For calculating a color temperature under various kinds ofenvironments of the external world, first, RGB signal intensities arecalculated using the white RGB signal gain under the reference colortemperature 6500K. Next, a color temperature can be acquired from theratio among these RGB signal intensities and the correspondingrelationship illustrated in FIG. 3. If the ratio between R/G and B/G is4:1, the color temperature is approximately 3500K according to the graphin FIG. 3. The ratio is used at this time in consideration of aninfluence of a bias applied to the whole RGB signal intensitiesdepending on a degree of brightness of image data, but the colortemperature can be acquired from RGB signal intensities as long as theluminance is adjusted in advance. The method described with reference toFIG. 3 is a method for calculating a color temperature from a ratioamong RGB signal intensities, and is merely an example of a method forcalculating a color temperature from image data.

FIG. 4 is a flowchart illustrating processing performed by the standardillumination environment processing unit 104 in the video see-throughHMD 101. In the following description, the processing by the standardillumination environment processing unit 104 will be described withreference to FIG. 4. A central processing unit (CPU) in the videosee-through HMD 101 reads out a required program and required data froma recording medium such as a read only memory (ROM) to execute theprogram, by which the processing illustrated in FIG. 4, and theprocessing illustrated in FIGS. 5 and 6 that will be described below arerealized.

In step S401, the illumination environment information extraction unit202 inputs captured image data from the imaging device characteristiccorrection unit 201 of the imaging unit 103. In step S402, theillumination environment information extraction unit 202 extractsillumination environment information that contains color temperatureinformation and luminance information from the input captured imagedata. The extracted illumination environment information is output tothe illumination environment information processing unit 206 togetherwith image identification information from the image identificationinformation source 203.

In step S403, the illumination environment information extraction unit202 combines or adds the image identification information supplied fromthe image identification information source 203 to the captured imagedata. The illumination environment information extraction unit 202 maycombine the image identification information to the captured image databy the electronic watermark technique, or may add the imageidentification information to a header of the captured image data. Thecaptured image data with the image identification information combinedor added thereto is output to the luminance conversion unit 204.

In step S404, the luminance conversion unit 204 converts a luminance ofthe captured image data. The captured image data after the luminanceconversion is output to the color conversion unit 205. In step S405, thecolor conversion unit 205 converts a color of the captured image data.The captured image data after the color conversion is output to theimage output unit 105 in the video see-through HMD 101.

Next, processing performed by the local illumination environmentprocessing unit 107 in the video see-through HMD 101 will be describedwith reference to FIGS. 5 and 6. FIG. 5 is a flowchart illustrating aflow of image data in the local illumination environment processing unit107.

In step S501, the image identification information extraction unit 207inputs mixed reality image data from the image input unit 106. In stepS502, the image identification information extraction unit 207 extractsimage identification information from the mixed reality image data. Themixed reality image data, from which the image identificationinformation is extracted, is output to the image storage unit 209together with the image identification information. Further, theextracted image identification information is also output to the timingadjustment unit 208.

In step S503, the image storage unit 209 stores the image identificationinformation and the mixed reality image data in association with eachother. In step S504, the luminance conversion unit 210 reads out themixed reality image data from the image storage unit 209. Morespecifically, in response to an input of the image identificationinformation and an image readout permission from the timing adjustmentunit 208, the image storage unit 209 makes mixed reality image datacorresponding to this image identification information ready for beingread out from the image storage unit 209. The luminance conversion unit210 reads out this display image data, once the mixed reality image databecomes ready for being read out from the image storage unit 209.

In step S505, the luminance conversion unit 210 converts a luminance ofthe mixed reality image data. The luminance conversion unit 210 uses theluminance information contained in the illumination environmentinformation from the illumination environment information processingunit 206 as an adjustment value. The display image data after theluminance conversion is output to the color conversion unit 211. In stepS506, the color conversion unit 211 converts a color of the displayimage data. More specifically, the color conversion unit 211 adjusts RGBsignals based on the color temperature information contained in theillumination environment information from the illumination environmentinformation processing unit 206. The display image data after the colorconversion is output to the display device characteristic correctionunit 212 of the display unit 108.

FIG. 6 is a flowchart illustrating timing adjustment processingperformed by the local illumination environment processing unit 107 ofthe video see-through HMD 101.

In step S601, the image identification information extraction unit 207extracts the image identification information from the mixed realityimage data. The extracted image identification information is output tothe timing adjustment unit 208. In step S602, the timing adjustment unit208 outputs the image identification information input from the imageidentification information extraction unit 207 to the illuminationenvironment information processing unit 206.

In step S603, the illumination environment information processing unit206 reads out illumination environment information corresponding to theimage identification information input from the timing adjustment unit208, from the storage unit thereof. The illumination environmentinformation processing unit 206 stores the image identificationinformation and the illumination environment information input from theillumination environment information extraction unit 202 in a statebundled as one set. If the illumination environment informationprocessing unit 206 succeeds in reading out the illumination environmentinformation, the illumination environment information processing unit206 outputs a readout success signal (ACK) to the timing adjustment unit208. The illumination environment information processing unit 206outputs the illumination environment information and the imageidentification information to each of the luminance conversion unit 210and the color conversion unit at the same time as the output of the ACK.

In step S604, the timing adjustment unit 208 determines whether an ACKis input. If an ACK is not input even after a time-out time period haselapsed (NO in step S604), the processing returns to step S602. Then,the timing adjustment unit 208 outputs image identification informationof mixed reality image data one frame before the current frame to theillumination environment information processing unit 206. On the otherhand, if an ACK is input within the time-out time period (YES in stepS604), the processing proceeds to step S605.

In step S605, the luminance conversion unit 210 outputs an image inputready signal to the timing adjustment unit 208 together with the imageidentification information at a timing at which it becomes possible toperform the luminance conversion processing using the illuminationenvironment information from the illumination environment informationprocessing unit 206.

In step S606, by being triggered by an input of the image identificationinformation and the image input ready signal, the timing adjustment unit208 transmits a permission for reading out mixed reality image datacorresponding to this image identification information to the imagestorage unit 209. The luminance conversion unit 210 reads out the mixedreality image data, once it becomes possible to read out the image fromthe image storage unit 209.

By the above-described processing from step S601 to step S606, theluminance conversion unit 210 and the color conversion unit 211 cansequentially perform the luminance conversion processing and the colorconversion processing using the illumination environment informationwith a delay as short as possible.

FIG. 7 is a time chart indicating a temporal relationship between a flowof image data and a flow of information in the local illuminationenvironment processing according to the first exemplary embodiment. Asillustrated in FIG. 7, according to the present exemplary embodiment,mixed reality image data is input into the image identificationinformation extraction unit 207, and image identification information isextracted from this mixed reality image data. Thereafter, the mixedreality image data is stored into the image storage unit 209. While themixed reality image data is stored into the image storage unit 209, thefollowing processing is performed. The image identification informationof this mixed reality image data is output from the timing adjustmentunit 208. Illumination environment information corresponding to thisimage identification information is read out by the illuminationenvironment information processing unit 206. Further, an ACK is inputinto the timing adjustment unit 208, and an image input ready signal isacquired by the timing adjustment unit 208. Then, after the mixedreality image data is stored into the image storage unit 209 and theimage input ready signal is acquired by the timing adjustment unit 208,the luminance and the color are converted by the luminance conversionunit 210 and the color conversion unit 211, respectively.

FIGS. 8A, 8B, 8C, and 8D illustrate data structures of the illuminationenvironment information. As illustrated in FIG. 8A, the illuminationenvironment information includes the image identification information(date/time information), the color temperature information, and theluminance information.

FIG. 8B illustrates an example of a data structure of the imageidentification information (the date/time information). As illustratedin FIG. 8B, the image identification information (the date/timeinformation) is information that indicates when captured image data iscaptured, and is mainly divided into date information and timeinformation. The date information includes year according to the Westerncalendar, month, and day. The time information includes hour, minute,second, and millisecond. FIG. 8B illustrates a structure example of 40bits in total.

The image identification information (the date/time information) onlyhas to be information that allows identification of image data from thetime when captured image data is captured to the time when virtual imagedata is superimposed onto this captured image data. Therefore, asdescribed above, if this time interval corresponds to 10 frames, theimage identification information only has to be information that allowsimage data pieces of the 10 frames to be identified, respectively.However, actually, the data structure of the image identificationinformation is determined in consideration of factors such as avariation in a time period required to generate virtual image data, anda possibility that the image identification information is also used asattribute information of the captured image data. For example, if theabove-described time interval during which image data should beidentified is not a constant interval from about 120 frames to about 180frames due to a variation in the time period required to generatevirtual image data, it is desirable that the image identificationinformation has a data structure having a margin, for example, a datastructure that allows image data pieces of 240 frames to be identified,respectively. If the image identification information is also used asattribute information of the captured image data, it is desirable thatthe image identification information has such a data structure thatattribute information such as a resolution, an image format, and ashutter speed is added to the data structure illustrated in FIG. 8B.FIGS. 8C and 8D illustrate examples of data structures of the colortemperature information and the luminance information, respectively. Thecolor temperature information is information that indicates a colortemperature itself, as illustrated in FIG. 8C. The luminance informationis information that indicates a Y signal when image data is expressed inthe YUV format, as illustrated in FIG. 8D.

FIGS. 9A, 9B, 9C, and 9D illustrate what kind of characteristics animaging device and a display device have. Representative examples of theimaging device characteristic include a receivable light frequencycharacteristic and a sensitivity characteristic.

FIG. 9A illustrates an example of the receivable light frequencycharacteristic of the imaging device. The characteristic of frequency oflight receivable by the imaging device varies for each imaging devicedue to influences of absorption and scattering of a color filter and aperipheral member used in the imaging device. This affects a tint ofcaptured image data.

FIG. 9B illustrates an example of the sensitivity characteristic of theimaging device. Suppose that the imaging device has a Bayer array, andmeasures signal outputs (VGr, VGb, VR, and VB) at a center of a screenwhen a white object is imaged. A gain of an electric signal is to beadjusted so that, if a light intensity of each RGB single color is thesame, an intensity of an electric signal after photoelectric conversionbecomes also the same for each RGB. This affects a luminance of capturedimage data.

The imaging device characteristic is not limited to the above-describedreceivable light frequency characteristic and sensitivitycharacteristic, and may be another element in the imaging unit 103 suchas an infrared (IR) filter and an analog transmission characteristic.The imaging device characteristic correction unit 201 corrects acharacteristic (a tint, a luminance, and the like) that depends on acomponent of the imaging unit 103.

Representative examples of the display device characteristic include anemittable light frequency characteristic and a luminance characteristicof a light source. FIG. 9C illustrates an example of the emittable lightfrequency characteristic of the light source. This example correspondsto emittable light frequencies of respective RGB colors whenlight-emitting diodes (LEDs) of RGB three colors are used as the lightsource. This affects a tint of mixed reality image data.

FIG. 9D illustrates an example of the luminance characteristic of thelight source. A current supplied to the LED is to be adjusted for eachRGB to adjust a light intensity of each RGB single color. This affects aluminance of mixed reality image data.

The display device characteristic is not limited to the above-describedemittable light frequency characteristic and luminance characteristic ofthe light source, and may be another element in the display unit 108such as a liquid-crystal filter characteristic of a display panel, and alens characteristic. The display device characteristic correction unit212 corrects a characteristic (a tint, a luminance, and the like) thatdepends on a component of the display unit 108.

In this manner, according to the present exemplary embodiment, mixedreality image data can be dynamically converted into image data thatmatches an illumination environment of an external world with use ofillumination environment information extracted from captured image data.Therefore, brightness and color sensations felt by an HMD user to anambient environment can substantially match an actual environment of theexternal world, which allows the user to be further engrossed into amixed reality space.

Further, captured image data output from the video see-through HMD 101into the image processing apparatus 102 is image data in which thecharacteristics depending on the imaging device are canceled, and abrightness and a white balance are adjusted. Therefore, the presentexemplary embodiment has advantages of compliance with a basic idea ofcolor matching, facilitation of marker extraction from captured imagedata to detect a position and an orientation, elimination of thenecessity of changing a color temperature and a luminance of the lightsource during drawing of virtual image data for each system, a reductionin loss in image data due to compression, and the like. Further, thecharacteristics depending on the display device are canceled from mixedreality image data in the display unit 108, which allows formation ofimage data according to a standard that does not depend on the displaydevice. Further, the present exemplary embodiment does not require anadditional sensor different from the components of the video see-throughHMD 101 itself, and utilizes a corresponding relationship betweencaptured image data and mixed reality image data, thereby simplifyingthe configuration.

Next, a second exemplary embodiment of the present invention will bedescribed. FIG. 10 is a block diagram illustrating a functionalconfiguration of a mixed reality system using a video see-through HMD1001 according to the second exemplary embodiment. Referring to FIG. 10,the mixed reality system includes the video see-through HMD 1001according to the second exemplary embodiment. The video see-through HMD1001 includes the imaging unit 103, the standard illuminationenvironment processing unit 104, the captured image storage unit 110,the image input unit 106, a time alignment unit 1004, the imagecombining unit 113, the local illumination environment processing unit107, the display unit 108, the position/orientation measurement unit109, an interface (I/F) unit 1003, and the image output unit 105.

Further, Referring to FIG. 10, the mixed reality system includes animage processing apparatus 1002. The image processing apparatus 1002includes the I/F unit 1003, the content DB 111, the CG drawing unit 112,and the image output unit 105. The image processing apparatus 1002 canbe embodied by an apparatus that has a high-performance calculationprocessing function and graphics generation/display function, such as apersonal computer and a workstation. The mixed reality system furtherincludes an external output apparatus 1005. This is a display apparatussuch as a plasma display panel (PDP), an organic electroluminescence(EL) display, and a liquid-crystal display. In FIG. 10, units similar tothose in FIG. 1 are labeled in a similar manner to FIG. 1.

According to the first exemplary embodiment, captured image data isoutput from the video see-through HMD 101 to the image processingapparatus 102. On the other hand, according to the second exemplaryembodiment, position/orientation information and image identificationinformation are output from the video see-through HMD 1001 to the imageprocessing apparatus 1002, and captured image data and the imageidentification information are bundled together to be stored as one setinto the captured image storage unit 110 of the video see-through HMD1001. The first exemplary embodiment has such an advantage that thevideo see-through HMD 101 can be simply configured, because the firstexemplary embodiment can be realized by using functions of an existingimage processing apparatus. On the other hand, the second exemplaryembodiment has such advantages that a communication amount can bereduced between the video see-through HMD 1001 and the image processingapparatus 1002, and a delay due to an output of image data can bereduced.

The image processing apparatus 1002 generates virtual image data fromthe position/orientation information, and outputs the virtual image datato the video see-through HMD 1001 together with the image identificationinformation. The image identification information is embedded in thevirtual image data by the electronic watermark technique, or is added toa header of the virtual image data. The captured image data and thevirtual image data are combined within the video see-through HMD 1001,and mixed reality image data, which is the combined image datatherefrom, is displayed on the display unit 108

Because captured image data and virtual image data are combined withinthe video see-through HMD 1001, time alignment processing foreliminating a time lag between the captured image data and the virtualimage data has to be performed in the video see-through HMD 1001.Therefore, according to the second exemplary embodiment, the timealignment unit 1004 is newly provided so as to allow the image combiningunit 113 to combine captured image data and virtual image data thatcoincides with the captured image data on a temporal axis. The timealignment unit 1004 performs time alignment processing for the localillumination environment processing unit 107, in addition to timealignment processing for the image combining unit 113.

According to the second exemplary embodiment, the video see-through HMD1001 includes the image combining unit 113, whereby the second exemplaryembodiment is configured in such a manner that mixed reality image datais held only by the video see-through HMD 1001. Further, mixed realityimage data is output from the image combining unit 113 to not only thelocal illumination environment processing unit 107 but also the imageoutput unit 105, to allow another person than an HMD user to alsoobserve the mixed reality image data. The mixed reality image data isoutput from the image output unit 105 to the external output apparatus1005. With this configuration, image data that does not depend on anobservation environment of the HMD user can be output to the externaloutput apparatus 1005 as mixed reality image data. Preparing mixedreality image data as image data according to the sRGB standard or thelike allows the external output apparatus 1005 to convert this data intoimage data suitable for its observation environment.

FIG. 11 is a block diagram illustrating a further detailed configurationof the video see-through HMD 1001 according to the second exemplaryembodiment. In FIG. 11, units similar to those in FIG. 2 are labeled ina similar manner to FIG. 2.

Referring to FIG. 11, the imaging unit 103 has a function similar to theimaging unit 103 illustrated in FIG. 2. The standard illuminationenvironment processing unit 104 includes the illumination environmentinformation extraction unit 202, the image identification informationsource 203, the luminance conversion unit 204, and the color conversionunit 205. The image identification information source 203, the luminanceconversion unit 204, and the color conversion unit 205 have functionssimilar to the image identification information source 203, theluminance conversion unit 204, and the color conversion unit 205illustrated in FIG. 2, respectively.

The illumination environment information extraction unit 202 extractsillumination environment information that contains color temperatureinformation and luminance information from captured image data byperforming image processing, and outputs the extracted illuminationenvironment information to the local illumination environment processingunit 107 together with image identification information from the imageidentification information source 203. In addition thereto, theillumination environment information extraction unit 202 adds the imageidentification information to the captured image data, and outputs thiscaptured image data to the luminance conversion unit 204. The presentexemplary embodiment is configured so as not to transmit the capturedimage data from the video see-through HMD 1001 to the image processingapparatus 1002. Therefore, it is desirable to handle the imageidentification information as different information from the capturedimage data instead of embedding the image identification informationinto the captured image data, in consideration that the imageidentification information is stored together with the captured imagedata as one set into the captured image storage unit 110, which is alater stage.

The captured image storage unit 110 stores image identificationinformation and captured image data corresponding to this imageidentification information as one set. The position/orientationmeasurement unit 109 extracts a marker and/or a natural feature from thecaptured image data input from the standard illumination environmentprocessing unit 104, and measures position/orientation information at aline-of-sight position of the HMD user. The position/orientationmeasurement unit 109 bundles the measured position/orientationinformation and image identification information corresponding to thisposition/orientation information together as one set, and outputs themto the I/F unit 1003. The I/F unit 1003 transmits theposition/orientation information and the image identificationinformation bundled together as one set to the image processingapparatus 1002.

The local illumination environment processing unit 107 includes theillumination environment information processing unit 206, the luminanceconversion unit 210, and the color conversion unit 211. In FIG. 2, theimage identification information extraction unit 207 and the timingadjustment unit 208 are provided as components of the local illuminationenvironment processing unit 107. In FIG. 11, they are provided inanother block as the time alignment unit 1004. The local illuminationenvironment processing unit 107 has a function and a processing contentsimilar to FIG. 2, only in terms of the local illumination environmentprocessing. However, the time alignment unit 1004 configured as anindependent block in FIG. 11 is extended so as to function even in timealignment processing for the image combining processing by the imagecombining unit 113.

The time alignment unit 1004 includes the image identificationinformation extraction unit 207 and the timing adjustment unit 208. Thetime alignment unit 1004 performs not only the time alignment processingfor the image combining unit 113 but also the time alignment processingfor the local illumination environment processing unit 107.

The image identification information extraction unit 207 extracts imageidentification information from virtual image data. This imageidentification information is the image identification information addedto the captured image data by the illumination environment informationextraction unit 202. The extracted image identification information isoutput to the timing adjustment unit 208. Further, the virtual imagedata is output to the image combining unit 113 together with theextracted image identification information.

The image combining unit 113 receives the output virtual image data andreads out captured image data corresponding to the image identificationinformation thereof from the captured image storage unit 110, and thencombines the virtual image data and the captured image data. Mixedreality image data after the combining processing is output to theluminance conversion unit 210 and the image output unit 105.

FIG. 12 is a time chart indicating a temporal relationship between aflow of image data and a flow of information in the image combinationprocessing and the local illumination environment processing accordingto the second exemplary embodiment. As illustrated in FIG. 12, at atiming when virtual image data and captured image data are combined bythe image combining unit 113, the following processing is performed.Image identification information of the mixed reality image data isoutput by the timing adjustment unit 208. Illumination environmentinformation corresponding to this image identification information isread out by the illumination environment information processing unit206. Further, an ACK is input into the timing adjustment unit 208, andan image input ready signal is acquired by the timing adjustment unit208. This is a difference from the first exemplary embodiment.

According to the present exemplary embodiment, mixed reality image dataoutput from the video see-through HMD 1001 into the external outputapparatus 1005 is image data in which the characteristics depending onthe device are canceled, and a brightness and a white balance areadjusted. Therefore, mixed reality image data according to a standardthat does not depend on the display device can be output. The presentexemplary embodiment complies with a basic idea of color matching,thereby having an advantage of being able to provide a display suitablefor an external environment and a device characteristic of the externaloutput apparatus 1005.

FIGS. 13A and 13B are block diagrams illustrating a modification of themixed reality system according to the second exemplary embodiment. Morespecifically, FIG. 13A illustrates a configuration of the modificationof the mixed reality system according to the second exemplaryembodiment, and FIG. 13B illustrates switching processing by a selector1301. The mixed reality system illustrated in FIG. 13A corresponds tothe configuration of the mixed reality system illustrated in FIG. 10with the selector 1301 added thereto.

For displaying only captured image data on the video see-through HMD1001 instead of displaying mixed reality image data, the selector 1301is switched to (2) illustrated in FIG. 13B, thereby changing a data flowso as to transmit image data from the imaging unit 103 to the displayunit 108. At that time, the mixed reality system can be switched to alow power consumption mode with the components other than the imagingunit 103, the selector 1301, and the display unit 108 substantiallystopping working. The imaging unit 103 performs processing that enablesan environment of captured image data to match an environment of anexternal world, and the display unit 108 displays the input mixedreality image data while maintaining its luminance and tint, therebyallowing an HMD user to observe an image that matches the environment ofthe external world. Further, for superimposing virtual image data ontocaptured image data, the selector 1301 is switched to (1) illustrated inFIG. 13B. In this case, the mixed reality system according to themodification has functions and processing similar to the mixed realitysystem according to the second exemplary embodiment illustrated in FIG.10.

In this manner, if only captured image data is displayed on the videosee-through HMD 101 without superimposing virtual image data thereon,mixed reality image data can be dynamically adjusted to image data thatmatches an illumination environment of an external world. Further, adelay in the processing of the system, a processing load, and powerconsumption can be reduced.

Other Embodiments

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions recorded on a storage medium (e.g., non-transitorycomputer-readable storage medium) to perform the functions of one ormore of the above-described embodiment(s) of the present invention, andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or more ofa central processing unit (CPU), micro processing unit (MPU), or othercircuitry, and may include a network of separate computers or separatecomputer processors. The computer executable instructions may beprovided to the computer, for example, from a network or the storagemedium. The storage medium may include, for example, one or more of ahard disk, a random-access memory (RAM), a read only memory (ROM), astorage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2013-074267 filed Mar. 29, 2013, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: acomputer-readable memory; and one or more processors that are coupled tothe computer-readable memory and that are configured to cause the imageprocessing apparatus to implement: an obtaining unit configured toobtain illumination environment information, which indicates anillumination environment of a real space, from image data imaged by animaging unit; a first conversion unit configured to convert the imagedata into standard image data based on the illumination environmentinformation; a generating unit configured to generate standard mixedreality image data by combining the standard image data and virtualimage data; a second conversion unit configured to convert, based on theillumination environment information, the standard mixed reality imagedata generated by the generating unit into mixed reality image datacorresponding to the illumination environment of the real space; and anoutput unit configured to output the mixed reality image datacorresponding to the illumination environment of the real space, whichis obtained by converting the standard mixed reality image data by thesecond conversion unit.
 2. The image processing apparatus according toclaim 1, wherein the obtaining unit obtains, as the illuminationenvironment information, color temperature information based on a ratioamong RGB signal intensities of the image data.
 3. The image processingapparatus according to claim 2, wherein the first conversion unitconverts, based on the obtained color temperature information, the RGBsignal intensities of the image data so that a color temperature of theimage data becomes equal to a predetermined color temperature of thestandard image data, and wherein the second conversion unit converts,based on the obtained color temperature information, RGB signalintensities of the mixed reality image data so that a color temperatureof the mixed reality image data becomes equal to the color temperatureof the image data.
 4. The image processing apparatus according to claim1, wherein the obtaining unit obtains, as the illumination environmentinformation, luminance information included in the image data.
 5. Theimage processing apparatus according to claim 4, wherein the firstconversion unit converts, based on the obtained luminance information,luminance of the image data so that the luminance of the image databecomes equal to a predetermined luminance, and wherein the secondconversion unit converts, based on the obtained luminance information,luminance of the mixed reality image data so that the luminance of themixed reality image data becomes equal to the luminance of the imagedata.
 6. The image processing apparatus according to claim 1, whereinthe obtaining unit further combines or adds identification informationof the image data with or to the image data, wherein the firstconversion unit converts the image data, which includes, as a result ofthe combining or the adding, the identification information of the imagedata, into the standard image data including the identificationinformation of the image data, wherein the generating unit generates thestandard mixed reality image data including the identificationinformation of the image data by combining the standard image dataincluding the identification information of the image data with thevirtual image data, and wherein the second conversion unit converts thegenerated standard mixed reality image data, which includes theidentification information of the image data, based on the illuminationenvironment information of which identification information correspondsto the identification information of the image data.
 7. The imageprocessing apparatus according to claim 6, wherein the identificationinformation contains information that indicates when the image data isimaged by the imaging unit.
 8. The image processing apparatus accordingto claim 1, wherein the one or more processors are further configured tocause the image processing apparatus to implement a measurement unitconfigured to measure a position/orientation of the imaging unit basedon a feature, which is extracted from the standard image data, of thereal space, wherein the virtual image data is image data generated basedon the position/orientation measured by the measurement unit.
 9. Theimage processing apparatus according to claim 1, wherein the imagingunit is mounted on a head mounted display apparatus.
 10. The apparatusaccording to claim 1, further comprising a first correction unitconfigured to perform, on the image data imaged by the imaging unit,correction processing of correcting a characteristic of the imagingunit, wherein the obtaining unit obtains the environment illuminationinformation from the image data on which the correction processing hasbeen performed by the first correction unit.
 11. The apparatus accordingto claim 1, further comprising a second output unit configured to outputthe standard mixed reality image data generated by the generating unit,before conversion processing by the second conversion unit is performedon the standard mixed reality image data.
 12. The apparatus according toclaim 1, further comprising a second correction unit configured toperform, on the mixed reality image data obtained by converting thestandard mixed reality image data generated by the generating unit andcorresponding to the illumination environment of the real space, secondcorrection processing of correcting a characteristic of a display uniton which the mixed reality image data is displayed.
 13. An imageprocessing method to be performed by an image processing apparatus, themethod comprising: obtaining illumination environment information, whichindicates an illumination environment of a real space, from image dataimaged by an imaging unit; converting the image data into standard imagedata based on the illumination environment information; generatingstandard mixed reality image data by combining the standard image dataand virtual image data; converting, based on the illuminationenvironment information, the standard mixed reality image data intomixed reality image data corresponding to the illumination environmentof the real space; and outputting the mixed reality image datacorresponding to the illumination environment of the real space.
 14. Anon-transitory computer-readable medium storing a program for causing acomputer to perform a method comprising: obtaining illuminationenvironment information, which indicates an illumination environment ofa real space, from image data imaged by an imaging unit; and convertingthe image data into standard image data based on the illuminationenvironment information; generating standard mixed reality image data bycombining the standard image data and virtual image data; converting,based on the illumination environment information, the standard mixedreality image data into mixed reality image data corresponding to theillumination environment of the real space; and outputting the mixedreality image data corresponding to the illumination environment of thereal space.